1. Field of the Invention
The invention generally relates to a device and a method for processing images, more specifically to a device and a method for automatically detecting a shadow in a supplied image, and also to a device and method for segmenting a supplied image into a shadow region and a non-shadow region.
2. Description of the Related Art
In a technical field of the color image analysis, segmenting an image into a shadow region and a non-shadow region or detection of the shadow region plays an important role, especially in white balance processing and exposure compensation processing in cameras. In the related art white balance processing, color is adjusted for an entire image; however, with this method, objects in shadows frequently appear to have blue casts. However, when photographing human subjects, the blue casts in human faces are generally considered unsightly and undesirable. Accordingly, attempts have been made to provide a technology to detect a shadow region of images in advance and provide different white balance parameters for a shadow region and for a non-shadow region, so that the blue casts may be prevented from appearing in photographed images of human subjects.
U.S. patent application serial number 2007/0110309A1 discloses a technology to detect shadows in images. In this technology, one or more images containing shadows are compared to reference images that are void of the shadows, and ratios are calculated between pixel pairs of the images. The pixels having the calculated ratios that fall within a predetermined range are detected as having shadows. However, in photographing or imaging subjects with a camera, it may be difficult for a user to provide reference images that are void of the shadows corresponding to all the scenes the user desires to take images of.
Further, U.S. Pat. No. 7,305,127 B2 discloses a technology to detect shadows in HSV color space properties, that is, hue (H), saturation (S), and brightness (V). This technology is devised based on the following hypothesis. According to U.S. Pat. No. 7,305,127 B2, in the H/S (ratio of hue to saturation) histogram, the valley of the H/S histogram indicating a near position of the value of 1 corresponds to a border between pixels in a shadow region and those in a non-shadow region of an image. The values located on the left of the boundary line of the histogram correspond to the pixels in the shadow region and the values located on the right of the boundary line of the histogram correspond to the pixels in the non-shadow region of the image. Likewise, in the S/V (ratio of saturation and brightness) histogram, the peak of the S/V histogram indicating a near position of the value of 1 corresponds to a border between pixels in the shadow region and those in the non-shadow region of the image. The values located on the left of the boundary line of the histogram correspond to the pixels in the non-shadow region and the values located on the right of the boundary line of the histogram correspond to the pixels in the shadow region of the image. However, such a hypothesis may not be supported if images are composed of complex scenes.
A pixel value of the image is a product of the light intensity and the reflectance of positions corresponding to the pixels. If x and y respectively represent horizontal and vertical coordinates, I(x, y) represents the pixel value of the pixels, and S(x, y) and R(x, y) respectively represent the light intensity and the reflectance of the pixels of the shadow region of the image, the relationship between I(x, y), S(x, y), and R(x, y) is represented by the following equation (1).I(x,y)=S(x,y)×R(x,y)  (1)In equation (1), an analytical value of light intensity S of the shadow region of the image may not be obtained from the supplied image I alone. Thus, according to equation (1), shadows may not be detected, in theory. However, the shadow region of the image can be discriminated with the naked eye, which means that some characteristic by which the shadow region is identifiable still exists in the image.
Notice that the white balance processing may be required to be carried out in real time with shadow detection. Accordingly, a shadow region and a non-shadow region of an image are discriminated based on color characteristics in the image. The shadow detection may particularly be effective when the density of the shadow is significant. That is, an image taken on a sunny day has a significant contrast between a shadow region and a non-shadow region of the image. By contrast, an image taken on a cloudy day has insignificant contrast between the shadow region and the non-shadow region of the image. Accordingly, the image taken on a cloudy day need not be divided into two regions in order to independently carry out white balance processing on the two regions. Examples of such two regions (i.e., shadow and non-shadow regions) may include an image taken on a sunny day and an image taken on a cloudy/rainy day; however, examples may include images taken on a variety of scenes such as an image taken indoor and an image taken outdoor.
Below, the images taken outdoor are described in more detail. The embodiments of the present invention disclose a novel and useful technology to detect shadows utilized in digital cameras. As mentioned above, in theory, the analytical value of light intensity of the shadow region of the image may not be obtained from the supplied image alone. However, statistically speaking, shadows may be identifiable to some extent based on their characteristics. An example of such characteristics may be darkness. Accordingly, in the following embodiments, a shadow detection method is devised based on a machine learning approach.
First, distributions of two types of image samples, namely, shadow image samples and non-shadow image samples are analyzed, and a collection of color characteristics based on which shadow pixels and non-shadow pixels can be identifiable is obtained. The collection of color characteristics includes brightness, saturation, and a ratio of a blue amount and a red amount in an RGB color space. Next, a classifier is created based on groups of image samples collected in advance. Elements supplied to the classifier may be unique vectors including values of the collection of color characteristics. The classifier may be configured to output attribute values (e.g., shadow and non-shadow). If an image contains a shadow region and a non-shadow region (typically a shadow region appears in an image of a sunny day), the contrast for the entire image is high. Accordingly, the machine learning approach is effective for detecting the shadow region of the image. By contrast, if the contrast of the image is low (e.g., few shadow regions appear in an image of a cloudy day), the shadow detection based on the machine learning approach may produce an inaccurate outcome.
In the following embodiments, a discriminating step in which images not taken on the sunny day (i.e., cloudy day or rainy day) are filtered out is provided before a classifying step in which the shadow and non-shadow regions of the images are classified. Since the mages not taken on the sunny day generally include few shadows, it may not be necessary to segment the image into two, namely, a shadow region and non-shadow region to independently carry out white balance processing on the two regions. Scenes identifying and classifying technologies often include complex texture characteristic amounts such as a directional histogram and a local binary pattern. Such characteristic amounts are considered to be too complex to be computed by the hardware environment of a camera.
In view of the hardware processing capacity of digital cameras and the requirement for white balance processing, the supplied image is divided into blocks of the same size, shadow detection is carried out on each block, and the digital camera in the following embodiments incorporates hardware to measure the mean brightness and color value of the blocks of a certain size.